v2.3.0 - #329
Merged
Merged
Conversation
Add Microsoft.DBforPostgreSQL/serverGroupsv2 (Citus distributed Postgres) as a first-class Azure service, SDK-compatible with armcosmosforpostgresql. - Driver interface (services/cosmospostgresql/driver): clusters, firewall rules, roles, derived servers/nodes, configurations, private endpoints/links. - In-memory provider (providers/azure/cosmospostgresql): clone-on-read/write, parent->child cascade delete, start/stop/restart lifecycle, read-replica linkage + promotion, derived coordinator/worker nodes, a server-parameter catalog with per-role overrides, checkNameAvailability. - ARM REST handler (server/azure/cosmospostgresql): PUT/PATCH return the resource inline with a terminal provisioningState; cluster actions reply 202 + Location and complete via operationStatuses. Generic serveCRUD/ serveReadOnly/armListOf helpers keep the parallel handlers DRY. - Wired into the Azure provider/server bundles + DriversFrom; cosmospostgresql cost key + rate test. - Provider unit tests + real-SDK round-trip tests (armcosmosforpostgresql). - Docs: services.md section 11e + master row 17d + counts; README; sdk-server.md.
…dation - Fix request-triggered crash: PATCH now re-validates nodeCount bounds in applyClusterPatch, and nodesForCluster clamps the make() cap. - Read-replica model: DeleteCluster unlinks both sides (orphans replicas / drops the deleted replica from its source); replica create validates the source exists and is a primary (no replica-of-a-replica chains). - Global cluster-name uniqueness on create, matching CheckNameAvailability. - Firewall rules validate IPv4 format and Start <= End. - Missing-parent handling is NotFound throughout (child creates + list ops for firewall rules / roles / private-endpoint connections), matching the config ops and real Azure. - Lows: all replica nodes read-only (coordinator included); start/stop/restart state guards; vCores/storage sizing validation; server-parameter value validation (reject empty, enforce enum/range AllowedValues); private-endpoint status validation + ActionsRequired default; role password required. - Tests: provider 63.6% -> 89.3%, server 60.2% -> 80.1%.
- High: sourceResourceId is now immutable on re-PUT — a re-PUT preserves the existing source instead of re-pointing it, so the replica graph can't be corrupted (no stale/self/chained links). - PATCH now applies coordinatorEnablePublicIpAccess / nodeEnablePublicIpAccess / enableShardsOnCoordinator (added to ClusterPatch) and accepts the write-only administratorLoginPassword. - Wire cluster numerics (nodeCount, vCores, storage) and maintenanceWindow fields are *int so a legitimate 0 serializes (single-node nodeCount=0 no longer deserializes to nil and PATCH-to-0 is no longer a silent no-op). - serverNames enumerates coordinator + workers, and both it and the servers sub-resource derive the FQDN from the cluster Location so they agree. - CreateOrUpdateCluster reports created-vs-updated so PUT returns 201 on create / 200 on re-PUT. - updateServerConfig returns NotFound for an unknown parameter (matches GetConfiguration); child Get/Delete check the parent cluster. - Tests: SDK promote LRO + restart-while-stopped (409), single-node nodeCount=0, serverNames/FQDN agreement, re-PUT immutability, PATCH writable fields, role-list assertion. Coverage provider 88.6% / server 81.6%.
- toARMServerConfiguration derives its `type` from the same collection segment as the `id` (coordinatorConfigurations / nodeConfigurations / configurations) so `type` and `id` agree. - checkNameAvailability test asserts the `nameAvailable` pointer is present (not just falsy) and adds a wire-level available==true case, so a dropped field would regress loudly.
Add Azure Cosmos DB for PostgreSQL Full-Parity Support
Give the cross-service inventory a uniform, resource-agnostic attribute set (SKU / Kind / ManagedBy / Zones / Properties) that every walker fills from its driver's fields and every row-builder renders the same way — no per-type or per-provider branching — so Azure Resource Graph (and AWS Resource Explorer / GCP Cloud Asset) emit the type-specific shape a real discovery + cost consumer reads. - resourcediscovery.Resource + DiscoveredDatabase/DiscoveredCluster gain the generic slots (via a shared Attributes struct on the projections). - Walkers populate the slots uniformly; a portable volume walker surfaces block volumes (EBS / Azure managed disks / GCE PDs) as first-class Volume resources with SKU=tier, Properties.diskSizeGB, and ManagedBy=owning instance. - Azure ARG resourceToWire renders sku/properties/managedBy/kind/zones when set, and derives id (ARM resource id) + resourceGroup from the id instead of the hardcoded "default". compute/Volume maps to microsoft.compute/disks (ARG + GCP Cloud Asset); AWS RE2 derives the type generically. - compute driver: Instance gains OSType/Priority/LicenseType/Zones; VolumeInfo gains IOPS/Throughput. Azure sqlDiscovery projects flexible-server SKU/storage/ HA and managed-instance SKU/vCores/storage through the generic slots. - Real-SDK acceptance test: seed a VM + attached Premium_LRS disk and assert the ARG rows carry sku.name, properties.diskSizeGB, managedBy, and ARM ids.
…server test - docs/sdk-server.md: document the resource-shape columns Resource Graph now emits (sku/properties/managedBy/kind/zones) and note the resourceGroup "default" fallback for mocks that don't model per-resource groups (+ Event Hubs still deferred), per review follow-up. - Add TestSDKResourceGraph_FlexServerFields: a PostgreSQL Flexible Server projects its compute SKU + HA mode through the same generic slots, proving the mechanism isn't VM/disk-specific.
Resource Graph: generic resource-shape attributes (VM SKU, managed disks, DB tier)
…ss-only IGW, endpoint services, Client VPN, Network Firewall Add the AWS-native networking resources cloudemu was missing, as AWS-only optional capability interfaces on the networking driver (like the existing NetworkInterfaces/VPCAttributes) plus a standalone Network Firewall service. No Azure/GCP stubs — each provider PR adds its own specifics. EC2 query protocol (providers/aws/vpc + server/aws/ec2), each behind an optional capability interface: - Transit Gateway: gateways, VPC attachments, route tables - VPN: customer gateways, VPN gateways (+attach/detach), VPN connections - DHCP option sets (+associate) - Managed prefix lists (+entries) - Egress-only internet gateways (IPv6) - VPC endpoint services (PrivateLink provider side) - Client VPN endpoints (+target-network association) Network Firewall (separate AWS JSON 1.0 service): Firewall, FirewallPolicy, RuleGroup — new services/networkfirewall driver, providers/aws/networkfirewall mock, server/aws/networkfirewall handler, wired into the AWS provider + server.
- Real-SDK round-trip tests: aws-sdk-go-v2 EC2 client across all new families (transit gateway, VPN, DHCP options, prefix lists, egress-only IGW, endpoint services, Client VPN) + aws-sdk-go-v2 networkfirewall client (firewall / policy / rule group). - Provider unit tests for the vpc capability families and the networkfirewall mock (validation + error paths); provider coverage vpc 84.7%. - docs/services.md: AWS-specific networking capability table + Network Firewall master-table row and summary; Grand Total 1415 -> 1462.
Add TGW routing operations: CreateTransitGatewayRoute, DeleteTransitGatewayRoute, SearchTransitGatewayRoutes, AssociateTransitGatewayRouteTable, and Enable/DisableTransitGatewayRouteTablePropagation (driver + provider + EC2 handler + real-SDK round-trip and provider unit tests). A transit gateway can now actually route between attachments, not just exist.
…rewall depth Adds depth operations across the AWS-only networking capabilities and Network Firewall so the shipped families reach full CRUD+behavioral parity: - VPN: VpnConnectionRoute Create/Delete, ModifyVpnConnection (+routes in describe) - Managed prefix lists: ModifyManagedPrefixList - VPC endpoint services: Modify/Describe permissions - Client VPN: DescribeTargetNetworks, Authorize/RevokeIngress, DescribeAuthorizationRules, Route Create/Delete/Describe - Network Firewall: AssociateFirewallPolicy, Associate/DisassociateSubnets, UpdateFirewallDeleteProtection, Update/DescribeLoggingConfiguration, Tag/UntagResource Extends the real-SDK round-trip tests and provider unit tests to cover the new operations, and updates docs/services.md counts.
…on, wire fidelity Resolves the PR review on the Stage A depth work: Thread safety - Add sync.RWMutex to networkfirewall.Mock (was unsynchronized); all methods now lock, fixing concurrent-map-writes on the logging map and unlocked pointer-field mutations. - Add RLock to every new vpc capability reader and Lock to the in-place mutators (transit-gateway / prefix-list deletes) that previously mutated shared pointers without holding the lock. - Add concurrent -race tests for both packages (the suite was sequential, so -race never exercised these paths). Wire fidelity - ModifyVpcEndpointServicePermissions returns <return>, not <returnValue> (matches the aws-sdk-go-v2 deserializer). - Add customerGatewayConfiguration to the VPN connection response. - Capture and echo Client VPN AuthenticationOptions; RuleGroup ARN uses stateless-rulegroup for STATELESS groups; correct the UnsuccessfulItem shape. Referential integrity - Validate gateway references on CreateVpnConnection/ModifyVpnConnection and attachmentId on CreateTransitGatewayRoute; enforce PrefixList MaxEntries; validate/guard Network Firewall policy references; in-use guards on DeleteTransitGateway and DeleteFirewallPolicy. Tests & docs - Cover the Delete/Describe/list halves and an SDK error path per service. - Correct docs/services.md op counts (58 AWS-specific, 20 Network Firewall, grand total 1493).
AWS networking parity: Transit Gateway, VPN, DHCP, prefix lists, egress-only IGW, endpoint services, Client VPN, Network Firewall
Adds the AWS IP Address Manager as a new optional EC2 capability (type-asserted on the vpc driver, matching the Stage A pattern): - IPAM: Create/Describe/Modify/Delete (creates a public + private default scope on create) - IPAM Scope: Create/Describe/Modify/Delete - IPAM Pool: Create/Describe/Modify/Delete - Pool CIDRs (supply): Provision/Deprovision/GetIpamPoolCidrs - Pool allocations (usage): Allocate/Release/Get/Modify Thread-safe from the start (RLock readers / Lock mutators) and validates references on create + guards in-use resources on delete (scope->ipam, pool->scope, allocation->pool, cidr provisioning). EC2 query-protocol wire element names verified against the aws-sdk-go-v2 deserializers. Advanced IPAM features (BYOASN, Resource Discovery, External Verification Tokens, Prefix-List Resolvers, Policies, Organization-Admin, discovered- resource scanning) are intentionally out of scope: they model multi-account Organizations with live network scanning, which an in-memory single-account emulator cannot represent. Documented in docs/services.md. Tests: real aws-sdk-go-v2 round-trip (TestEC2IPAMParitySDK), provider unit test with error paths, and a concurrent -race test. Docs op counts updated (AWS-specific networking 58->78, grand total 1493->1513).
…er, policy, metrics Extends the IPAM core lifecycle to the complete service surface (~49 more operations + the AWS/IPAM CloudWatch metrics layer), each a distinct optional EC2 capability type-asserted on the vpc driver: - Resource CIDRs + address history (derived from stored VPCs/subnets) - Resource Discovery: CRUD + associations + discovered accounts/resource-cidrs/ public-addresses (a default resource discovery is created per IPAM) - BYOASN: provision/deprovision/associate/disassociate/describe - BYOIP + public IP: move-to-ipam/provision/deprovision/describe/advertise/withdraw - Prefix-List Resolver: resolvers + targets + versions/rules/version-entries - External Resource Verification Token: create/delete/describe - Policy + Organization admin: create/delete/describe/enable/disable/get-enabled/ allocation-rules/org-targets + enable/disable org-admin account Metrics: the vpc driver now exposes an optional IPAMMetrics capability that the CloudWatch handler surfaces under the AWS/IPAM namespace (ListMetrics + GetMetricStatistics): TotalActiveIpCount, pool Percent*/*ResourceCidrs, scope Managed/Unmanaged/Overlapping/*ResourceCidrs, public-IP insight counts, and VpcIPUsage/SubnetIPUsage — all computed live from IPAM + VPC/subnet/EIP state. Cross-account/organization and live-network features are modeled against the emulator's own single-account state (discovered resources derive from stored VPCs/subnets/EIPs; org targets resolve to the configured account). Thread-safe from the start (RLock readers / Lock mutators) with FK + in-use guards (scope->ipam, pool->scope, resolver->target, resource-discovery->assoc, advertised BYOIP not deprovisionable). Wire element names verified against the aws-sdk-go-v2 deserializers. Tests: real aws-sdk-go-v2 EC2 round-trip across every group (TestEC2IPAMFullSDK), a CloudWatch SDK metrics test (TestIPAMMetricsSDK), and a provider unit + -race test (TestIPAMFullProvider). Docs updated: AWS-specific networking 78->127, grand total 1513->1562, plus an AWS/IPAM metrics section.
The CloudWatch handler only matched rpc-v2-cbor requests, so the AWS CLI —
which sends CloudWatch as the classic query protocol (form-encoded POST,
Action=..., XML responses) — was stolen by the EC2 handler and every op
returned InvalidAction.
Add a query-protocol path to the CloudWatch handler, disambiguated from EC2
by the SigV4 credential scope service ("monitoring"). Implements PutMetricData,
ListMetrics, GetMetricStatistics, PutMetricAlarm, DescribeAlarms, DeleteAlarms,
and SetAlarmState (the last previously a gap). PutMetricData now defaults an
absent timestamp to now (so GetMetricStatistics returns datapoints).
Verified end-to-end with the real aws CLI; EC2 form-POST routing unaffected.
Adds TestQueryProtocol regression test.
SNS Publish stored the message but never delivered it to subscribers, so the SNS->SQS fan-out pattern silently dropped messages. Add an SQSDeliverer hook (satisfied by the SQS mock's new DeliverExternal, which enqueues by queue ARN) and fan out on Publish to every sqs-protocol subscription, wrapping the payload in the standard SNS Notification envelope. Wired via SetSQSDeliverer in aws.go. Verified end-to-end with the real aws CLI (subscribe + publish + receive). Adds TestSNSToSQSDelivery.
PutEvents matched rules against patterns but never propagated events to their targets, so SQS-target subscribers received nothing. Wire an injected SQSDeliverer and deliver the standard EventBridge event envelope to every matched target whose ARN is an SQS queue.
Describe{Vpcs,Instances,Volumes,SecurityGroups} with an explicit ID that
does not exist returned an empty success instead of the resource-specific
Invalid*.NotFound error real EC2 emits. That silently broke existence
checks, Terraform refresh/drift detection, and wait-until-deleted polls.
Providers now return NotFound for a missing explicit ID; the server
handlers already map it to InvalidVpcID/InvalidInstanceID/InvalidVolume/
InvalidGroup.NotFound.
The provider already implemented these, but the JSON-RPC handler didn't dispatch them, so callers couldn't read a queue's ARN (required for DLQ wiring, Lambda event-source mappings, and S3->SQS notifications), resize a queue, or drain it. Wire the three operations through to the existing driver methods.
UpdateFunctionConfiguration, PublishVersion/ListVersionsByFunction, and
the alias lifecycle (create/get/list/update/delete) returned 404
"unsupported Lambda path" — the driver implemented them but the REST
handler only dispatched the collection, resource, and invoke shapes. Add
routing for /{name}/configuration, /{name}/versions, /{name}/aliases, and
/{name}/aliases/{alias}. Resource policies (AddPermission), tagging, and
event-source mappings remain follow-ups.
Terraform's aws_lambda_permission and the invoke grants S3/SNS/
EventBridge create were unreachable (404). Add a per-function resource-
policy store in the AWS provider and route /{name}/policy(/{sid}).
Resource policies are Lambda-specific, so the handler type-asserts an
AWS-local policyManager rather than widening the portable Serverless
driver (which Azure Functions and GCP Cloud Functions also implement).
The emulator stores statements without evaluating them.
docker login and image push/pull authenticate via GetAuthorizationToken, which was unimplemented. The AWS ECR provider now returns a base64 "AWS:<password>" credential, the registry proxy endpoint, and a 12h expiry. Registry auth is ECR-specific, so the handler type-asserts an AWS-local authTokenProvider rather than widening the shared ContainerRegistry driver (Azure ACR and GCP Artifact Registry also implement it).
#319) UpdateSecret (routine metadata/value change) and secret tagging were unimplemented. Add UpdateSecret (description + optional new value version) plus Tag/Untag to the AWS provider, routed via an AWS-local secretMutator assertion so the shared Secrets driver (also implemented by Azure Key Vault and GCP Secret Manager) stays untouched.
CreateTags/DeleteTags returned InvalidAction, so every IaC tool that tags EC2 resources (nearly all of them) failed. Route the calls by resource-ID prefix: VPC-family IDs to the networking provider's existing tag methods, and instance/volume/snapshot/image IDs to a new AWS-local compute tagger. Unknown IDs return InvalidID.NotFound.
SNS TagResource returned InvalidAction, blocking IaC tag-on-create flows. Add TagTopic/UntagTopic to the AWS provider, routed via an AWS-local topicTagger assertion (the shared Notification driver is also implemented by Azure Notification Hubs and GCP FCM). ListTagsForResource is deferred: its action name collides with RDS in the query protocol and needs SigV4 credential-scope routing to disambiguate.
…s) (#319) The Lambda tagging API lives at the /2017-03-31/tags prefix, which the handler didn't match, so requests fell through to the S3 catch-all and returned a 405 + HTML body the SDK couldn't deserialize. Match that prefix and route POST/DELETE/GET to new AWS-local function-tagger methods.
SQS tag operations returned UnknownOperationException. Add queue tagging to the AWS provider (stored on QueueInfo.Tags), routed via an AWS-local queueTagger assertion so the portable MessageQueue driver stays untouched.
Tags could only be set at create time; AddTags/RemoveTags returned InvalidAction. Add tag-mutation methods to the ELB provider (updating the load balancer or target group by ARN) routed via an AWS-local tagMutator assertion. The empty <FooResult/> wrapper is included so the SDK deserializes the response.
…319) Route 53 tagging at /2013-04-01/tags/{type}/{id} fell through to the S3 catch-all. Match that prefix and route the two operations to a generic ID-keyed tag store on the provider via an AWS-local resourceTagger assertion.
…319) PutRule/DescribeRule/ListRules returned 'arn:aws:events:::rule/<bus>/<name>' with empty region and account. Thread accountID/region into the handler so rule ARNs are complete (arn:aws:events:<region>:<account>:rule/...).
CreateCluster left Version empty when the caller omitted it, so Create/DescribeCluster returned a null version. Default to the latest supported version (1.29), matching real EKS.
…coping, validation Resolves the review on #318: - cloudwatch ListMetrics: an empty ("list all") namespace no longer routes into the IPAM shortcut and drops every real metric. It now enumerates all real metrics with their true namespaces (new AWS-local ListMetricsDetailed capability) and merges the AWS/IPAM set; an exact AWS/IPAM request stays IPAM-only. - Provision/AllocateIpamPoolCidr: netmask-only requests (the standard AWS pattern) now derive a concrete, aligned CIDR from the pool's supply / default base via a sequential carve, instead of storing an empty CIDR that corrupted AWS/IPAM PercentAllocated/Available. - ModifyIpamResourceCidr: scope-move / unmonitor now persist via a per-resource override map applied in ipamResourceCidrs(), instead of mutating a throwaway copy under an RLock and silently reverting on the next read. - CreateIpamPrefixListResolverTarget: validates prefixListID against the managed-prefix-list store. - Associate/DisassociateIpamByoasn: associate validates the CIDR is provisioned; disassociate validates the ASN. Regression tests extended in networking_parity_test.go and the IPAM metrics SDK test.
…gaps (#319) - eventbridge: fold PutEvents index into generateEventID so byte-identical events in one call get unique ids under FakeClock - s3: notify on CompleteMultipartUpload/CopyObject and emit ObjectRemoved:Delete on DeleteObject; make the ObjectRemoved test actually delete and assert - lambda parity: mirror the no-handler 200+echo invoke stub to Azure Functions and GCP Cloud Functions - rds: scope-gate AddTags/RemoveTags/ListTags Matches to the rds SigV4 scope - sns: carry MessageAttributes through Publish -> SQS envelope - ecr: implement Set/Get/DeleteRepositoryPolicy - ec2: lint hygiene (stateRunning const, receiver/wsl cleanups) and a CreateNetworkInterface unknown-SubnetId negative test
Fix AWS Parity Audit Bugs: Event Delivery, Tagging, Describe NotFound & More (#319)
…services (#321) High-severity (silent-wrong-data reads, broken flows, missing surface): - compute: echo networkInterfaces on instance read; image-from-disk (no phantom instance); operations list - networks: persist+echo firewall rules; autoCreateSubnetworks; subnet parent-network link - pubsub: subscription registry — distinct sub/topic names, ackDeadline/labels round-trip, effective delete, publishTime - monitoring: real alert-policy conditions/combiner/enabled/labels round-trip + alertPolicies.patch - clouddns: delete+add of the same rrset no longer AlreadyExists; unique change id; dnsName FQDN - iam: SA getIamPolicy/setIamPolicy, signBlob/signJwt/generateAccessToken, enable/disable - artifactregistry, eventarc: /operations/{op} routing so GAPIC .Wait() resolves (was 404) Medium fidelity: - artifactregistry: non-DOCKER format + description; eventarc: serviceAccount + labels - gcs: bucket PATCH wiring versioning + labels - cloudsql: getBackupRun surfaces real errors instead of masking as 404 - alloydb: report the driver's real lifecycle state (not hardcoded READY) - gke: apply desiredMaster/NodeVersion on UpdateCluster - cloudfunctions: emit httpsTrigger.url; add :generateUploadUrl - firestore: CreateDocument returns ALREADY_EXISTS on explicit-id collision - memorystore: memorySizeGb/redisVersion/displayName round-trip + instances.patch (Update) - cloudlogging: severity/jsonPayload/labels/insertId round-trip + orderBy desc - secretmanager: create makes an empty container (no phantom seeded version) - vertexai: endpoint :countTokens + stream routing - loadbalancer: backendService description/portName/healthChecks round-trip Each fix is backed by a real-SDK regression test. GCP fields the portable drivers can't model are round-tripped via reserved tags or handler-local state, keeping the shared driver interfaces (and the AWS/Azure providers) untouched.
…oken flows (#321) Standalone-server E2E (driving cloudemu serve --providers gcp with all handlers registered) surfaced bugs the single-driver package tests can't see: - Shared location-operations handler (server/gcp/lro): owns /v1/projects/{p}/locations/{l}/operations/{op}, registered first, returning a done operation (superset: longrunning `done` + GKE `status`). Fixes artifactregistry/eventarc/memorystore operation polls that alloydb/gke previously shadowed with a 404. - compute: instances.delete now removes the instance (GET-after-delete is 404) via a GCP-local hard-delete, instead of leaving a TERMINATED tombstone. - firestore: writes auto-create the collection (real Firestore behavior); a first document write no longer 404s "collection not found". - gcs: Matches no longer swallows unclaimed API-version paths (/v1, /sql, …) as bogus bucket lookups. - cloudlogging: entries:list sorts by timestamp for orderBy (was reverse-only, mis-ordering out-of-order writes). - pubsub: createTopic tolerates an empty body; networks: duplicate network create returns 409. Adds server/gcp/fullserver_test.go — a full-server harness exercising the collision paths so these regressions are caught in-suite.
- fcm: reject a message setting more than one target (token/topic/condition) — real FCM returns INVALID_ARGUMENT. - cloudasset: searchAllResources honors assetTypes and both search/assets.list emit nextPageToken so paged callers don't silently miss truncated results. (bigtable's InvalidArgument-on-missing-parent is intentional and tested, so it is left as-is; searchAllIamPolicies stays empty — it needs a cross-service IAM policy index the engine doesn't expose.)
…& cross-pool overlap (#318) - ModifyIpamPolicyAllocationRules: emit <ipamPolicyDocument> (the SDK output field) instead of <return>true>, which a real aws-sdk-go-v2 caller reads as IpamPolicyDocument == nil. - GetIpamPolicyAllocationRules: model allocation rules structurally (AllocationRules []IpamAllocationRule{SourceIpamPoolID} + Locale/ResourceType on the policy) so the response carries the real IpamPolicyDocument shape (allocationRuleSet>item>sourceIpamPoolId, ipamPolicyId) plus nextToken — previously each item deserialized empty. Request parses AllocationRule.N. SourceIpamPoolId + Locale + ResourceType. - ProvisionIpamPoolCidr: netmask-only provisioning now excludes CIDRs already provisioned by ANY pool sharing the carve base, so two pools no longer both receive 10.0.0.0/16 (cross-pool overlap defeated IPAM's non-overlap guarantee). Real-SDK regression test for the Modify/Get round-trip; provider test for the two-pool non-overlap.
…setter The round-1 change made cloudwatch server New take (monitoring, ipam), but development independently added a query-protocol test calling New(monitoring) with one arg. When the PR is test-merged into development the tree no longer compiles (go vet: not enough arguments in call to cwserver.New). Revert New to a single argument and attach the optional AWS/IPAM metrics source through a SetIPAMMetrics setter — matching the SetMonitoring/SetSQSDeliverer injection pattern used elsewhere — so both callers compile and the merged tree builds. No behavior change: IPAM metrics are still surfaced when the VPC driver implements the capability.
Resolve the CloudWatch handler conflict: keep development's query-protocol Matches (classic query + SigV4 monitoring-scope disambiguation) alongside this branch's IPAM-metrics support (ipam field + SetIPAMMetrics setter). Single-arg New is retained so development's query-protocol test compiles.
…ing tests (#321) Blockers: - monitoring alertPolicies.patch: decode enabled as *bool so an omitted field leaves it unchanged instead of silently disabling the policy. - pubsub subscriptions.list: read the subscription registry (distinct name + ackDeadline/labels) instead of emitting one phantom sub per topic queue. - artifactregistry/eventarc: emit the LRO response with an @type so a GAPIC apiv1 .Wait() can Any-decode it; add real apiv1 CreateX(...).Wait() tests. - monitoring: round-trip non-threshold conditions (conditionAbsent / MQL / PromQL / matchedLog) instead of modelling only conditionThreshold. Test coverage the earlier claim overstated: - gke Update now asserts desiredMasterVersion/desiredNodeVersion apply. - loadbalancer asserts description/portName/healthChecks round-trip. - cloudfunctions asserts httpsTrigger.url + generateUploadUrl. Low: - gcs Matches only reserves whole-segment API versions (v1, v1beta4) — a bucket like "v2-assets" is no longer false-rejected on the direct-media path. - fix "analogue" misspell in eventarc comments. Deferred (documented, non-blocking): monitoring/pubsub project-scoped keying (monitoring keyed by displayName), compute network-only NIC network:"" echo.
AWS Networking (Stage B): VPC IPAM (Full Parity + Metrics)
…#306) * feat(databricks): remaining Microsoft.Databricks ARM resources (#209) Finish the Microsoft.Databricks ARM control-plane surface beyond workspaces (built in #164), all reachable over the real armdatabricks SDK: - Access Connectors (accessConnectors): createOrUpdate, get, update, delete, list by resource group, list by subscription. System-assigned identities get synthesized principal/tenant IDs. - Private Endpoint Connections (workspaces/{w}/privateEndpointConnections): create, get, list, delete. - Private Link Resources (workspaces/{w}/privateLinkResources): get, list (synthesized databricks_ui_api / browser_authentication group set). - VNet Peering (workspaces/{w}/virtualNetworkPeerings): createOrUpdate, get, list, delete. - Outbound Network Dependencies Endpoints (workspaces/{w}/outboundNetworkDependenciesEndpoints): list (bare-array response, matching the SDK deserializer). - Operations (/providers/Microsoft.Databricks/operations): list — served via a subscription-less path special-case since azurearm.ParsePath requires a /subscriptions prefix. Built across all four layers: driver interface + types, in-memory provider Mock (memstore-backed, copy-on-write), portable service with the do() cross-cutting pipeline, and the SDK-compat ARM HTTP handler (routing extended for the new top-level type, workspace sub-resources, and the operations path). Modeled store-and-echo: the ARM resources round-trip faithfully over the SDK, but the underlying Azure networking side effects (real private endpoints, live VNet peering, outbound reachability) are not simulated — see docs/services.md. Tests: 21 SDK round-trip tests driving the real armdatabricks clients against an httptest server, plus 9 provider-level unit tests, covering happy paths and edge cases (not found, missing parent workspace, empty list, PATCH semantics, rejected PEC status, the no-subscription operations path). * fix(databricks): address review — case-insensitive routing, idempotent delete Addresses the re-review of #306: - Case-insensitive ARM routing (Medium): the handler matched the provider namespace and resource-type/sub-resource segments with ==, but ARM treats them case-insensitively (and the same file's isOperationsPath / the shared parseResourceGroup already use EqualFold). A lowercased path such as .../providers/microsoft.databricks/accessconnectors would 404. Now uses strings.EqualFold throughout Matches/ServeHTTP/serveWorkspaceChild. - Idempotent DELETE (Medium): DeleteAccessConnector / DeletePrivateEndpoint / DeleteVNetPeering returned 404 on a missing resource and 200 on success. Real ARM DELETE is idempotent — now 204 on success and 204 on a NotFound (matching the resourcegroups handler precedent), so teardown retries / delete-then-delete succeed. Pre-existing workspace delete (#164) left untouched. - Access-connector identity realism (Low): system-assigned principalId is now keyed on (resourceGroup, name) so same-named connectors in different RGs get distinct principals; tenantId is a single fixed emulator-wide directory GUID rather than a per-name synthesized value. - Docs (Low): §21 op-count total and the coverage-summary row bumped 52 -> 70 (this PR adds 18 driver operations). Tests: new SDK round-trip tests for case-insensitive routing (raw HTTP, since the SDK emits canonical casing), idempotent delete-of-missing across the three resources, and the PATCH-identity=None transition; all via the real armdatabricks clients. go build/vet/test/-race and golangci-lint (0 issues) green. * chore: re-trigger CI The prior CI run failed only on an unrelated, pre-existing flaky test: server/aws/bedrock TestSDKConverseStream ('use of closed network connection') — a load-sensitive httptest streaming-connection race from #298, not touched by this PR. All other checks pass and the test is green locally (30/30, incl. -race). No admin rights to re-run the job, so this empty commit re-triggers the pull_request workflow. * fix(bedrock): drain request body before streaming converse-stream response CI's `go test -race` intermittently failed TestSDKConverseStream with "use of closed network connection" — a pre-existing, load-sensitive flake (from #298) surfaced by the race detector's slowdown on the 2-core runner. Root cause: converseStream decodes the request via json.Decoder, which stops at the end of the JSON value and leaves the body unread (e.g. a trailing newline). With an unread request body, net/http cannot finish the connection gracefully once the chunked event-stream response has started, so it tears the TCP connection down when the handler returns — racing the SDK client's in-flight read of the stream. invokeModelStream already reads the whole body via io.ReadAll, which is why only converse-stream flaked. Fix: io.Copy(io.Discard, r.Body) after decoding, before switching to the streamed response, so the connection is finished cleanly and the client reads a normal end-of-stream instead of a reset. Full bedrock suite green under -race (20x on the streaming tests); build, vet, go test ./..., and golangci-lint clean.
…lic Access parity Extend the EC2 query-protocol handler with three EC2-family networking services as optional, type-asserted driver capabilities: - Traffic Mirroring: targets, filters, filter rules (+ network services), and sessions (Create/Delete/Describe/Modify). - Network Insights: Reachability Analyzer (paths + analyses) and Network Access Analyzer (access scopes, scope analyses, content, findings). - VPC Block Public Access: account/region options and per-VPC/subnet exclusions. Each ships all three layers (driver capability interface, in-memory VPC mock, query-protocol wire handler) with provider unit tests and real aws-sdk-go-v2 round-trip tests.
…to feat/gcp-parity-audit
subscriptions.list iterated the h.subs map directly, so with two or more subscriptions the response order varied run-to-run. Collect the names, sort them, and emit in order — matching the repo's deterministic list pillar. Extend the SDK test to two subscriptions (created out of sort order) and assert the sorted result.
Fix GCP Parity Audit Findings Across 19 Services
…nism, error fidelity Review fixes for Traffic Mirroring / Network Insights / VPC Block Public Access: - GetNetworkInsightsAccessScopeAnalysisFindings now emits the real AccessScopeAnalysisFinding object shape (populated from the driver's returned findings) instead of a string list that could never round-trip. - Timestamps (CreatedDate/StartDate/LastUpdateTimestamp) use opts.Clock.Now() so FakeClock keeps output reproducible. - DeleteTrafficMirrorTarget/Filter return DependencyViolation while a session still references them. - Per-resource *.NotFound error codes threaded into each op; missing referenced resources return NotFound (not InvalidParameterValue), and ModifyTrafficMirrorSession re-validates its target/filter like Create. - Describe* no-filter lists sort via SortedValues (deterministic order). - Get* reads hold the RLock; CreateNetworkInsightsPath requires a destination; VPC BPA block/exclusion modes are validated. - SDK round-trip tests for the Describe list handlers, resource-specific NotFound codes, DependencyViolation, findings shape, and enum rejection. - docs/services.md: add the three capability rows and bump the AWS-specific operation total (127 -> 162).
The re-pointed target/filter existence check in ModifyTrafficMirrorSession (added alongside the delete-dependency guard) had no test. Assert that modifying a session to a nonexistent target or filter returns NotFound, and that a non-re-pointing modify preserves the existing refs.
converseStream decoded the request with a json.Decoder, which stops at the end of the JSON value rather than reading to EOF. Returning from the handler with a half-read body makes net/http reset the connection instead of closing it cleanly, and the SDK eventstream reader observes "use of closed network connection" instead of io.EOF — an intermittent failure of the streaming round-trip tests under CI's -race load (invokeModelStream was unaffected because it already io.ReadAll's the body). Drain the remaining body with io.Copy(io.Discard, r.Body) before streaming so the connection terminates cleanly.
Resolve streaming.go: development independently added the same request-body drain fix for the bedrock eventstream connection-reset flake; keep the upstream comment over the shared io.Copy(io.Discard, r.Body).
Add Traffic Mirroring, Network Insights & VPC Block Public Access Parity
…cross all supported types (#323) (#327) * feat(resourcediscovery): project Azure Resource Graph cost fields across all supported types Emit the properties/sku a real armresourcegraph discoverer prices on — for VM/disk/VMSS, public IP, VNet/subnet, storage account, Cosmos DB, SQL MI/database/server + MySQL/Postgres flex, AKS cluster/agent-pool, Databricks, and App Service plans. Adds SKUTier/SKUCapacity slots to the generic attribute set and fixes two type-map bugs (NIC, public IP emitted wrong Azure types). New driver capabilities: Databases (Azure SQL logical DBs), ScaleSets (VMSS), AppServicePlans (serverfarms), and the optional BucketAttributes/TableAttributes capabilities that enrich storage accounts and Cosmos DB. All additive and nil-safe; AWS/GCP walks are unaffected. Backed by real-SDK ARG round-trip tests plus provider-unit tests. Extends #315/#316; addresses #323. * test(resourcegraph): consolidate ARG cost-field tests into one file Merge the four parallel-authored arg_cost_*_test.go files into a single arg_cost_fields_test.go with one shared helper set; assertions unchanged. * feat(server/azure): wire ARM create paths so cost fields survive create->discover Extend the Azure SDK-compat handlers to parse the cost fields off the ARM create body onto the driver config (they were accepted-and-dropped), and add create routes for resource types that had none: - virtualmachines: VM priority/licenseType/osType/zones; new VMSS handler - azuresql: MI storageAccountType; SQL Database create/get/list -> Databases - aks: cluster sku.tier; agent-pool scaleSetPriority - functions: new serverfarms (App Service plan) handler - storageaccount, cosmosaccount: new Microsoft.Storage/DocumentDB account handlers Real-SDK create->get round-trip tests per package. * feat(server/azure): finalize storage/cosmos account ARM create handlers Complete the Microsoft.Storage/storageAccounts and Microsoft.DocumentDB/ databaseAccounts create+get routes so their cost fields (sku/kind/accessTier, capabilities/offerType/enableFreeTier) survive create->discover. * feat(server/azure): wire disk ARM create fields + stamp configured subscription - disks: parse diskIOPSReadWrite/diskMBpsReadWrite/tier onto VolumeConfig; echo sku.tier. - resourcegraph: stamp the configured subscriptionId on every row instead of parsing inconsistent placeholders from each mock's ARN, so subscription-scoped ARG queries return the emulator's resources. * feat(server/azure): add publicIPAddresses ARM create/get route Wire Microsoft.Network/publicIPAddresses to AllocateAddress with sku + allocation method, so public IPs are creatable over the real SDK and their cost fields surface in Resource Graph. * fix(resourcediscovery): nest VM osType under storageProfile.osDisk to match real ARG VMs projected osType as a flat properties.osType; real Azure Resource Graph (and the VMSS projection) nest it under properties.storageProfile.osDisk.osType, so a real discoverer reading that path found it empty. Only Azure VMs set OSType (AWS/GCP mocks leave it blank), so no cross-provider shape leaks. * refactor(server/azure/functions): make createPlan/getPlan free functions (unused receiver) * fix(resourcediscovery): address ARG cost-field review findings - nonEmptyProps prunes empty/nested-empty maps and nil (keeps faithful bools); doc aligned to behavior. - Propagate BucketAttributes/TableAttributes errors instead of swallowing them. - Dedicated ServiceAppService discriminator for serverfarms (wire type unchanged). - Compile-time capability assertions for the optional discovery interfaces. - flexTier known-incomplete comment; unmask Databricks sku.tier assertion; add SQL-server version + Postgres-flex symmetry tests; §19 doc corrections.
… policy) — closes #312 (#314) * feat(k8s): dry-run, event field selectors, pod logs/exec, job shrink fix - server-side dry-run (?dryRun=All) echoes without persisting across registry + all typed writes - event field selectors (involvedObject.*, reason, type) - pods/log synthetic output; exec/attach/portforward return typed 501 - reconcileJob reconciles to exact completions (fixes overstated status.succeeded); surface pod-count clamp via annotation - AKS fallback kubeconfig advertises shared CA (provider CA symmetry) - README: correct stale k8s non-goals line * feat(k8s): deterministic clock, pagination, watch resume, finalizers - injectable config.Clock threaded through all data-plane timestamps (APIServer.SetClock); FakeClock makes creationTimestamps deterministic - list pagination (?limit=&continue=) across registry + typed list kinds - watch resourceVersion resume (skip snapshot replay) + BOOKMARK events - finalizer-gated deletion (deletionTimestamp; GC when finalizers drained) on the registry path and typed Namespace/Pod * feat(k8s): CustomResourceDefinition support (apiextensions.k8s.io/v1) - dynamic registry stores: RWMutex-guarded addStore/removeStore/getStore/allDefs - CRD reconcile materializes a servable store per served version; onDelete deregisters + cascade-deletes the custom resources - discovery repointed at the live registry so CRD kinds surface immediately (OpenAPI stays cluster-independent/static) - CRD status marked Established/NamesAccepted * feat(k8s): ReplicaSet interposition, DaemonSet nodeSelector, CronJob tick - Deployments now own a ReplicaSet per pod-template revision (Deployment→RS→Pod), matching real topology; rolling updates create a new RS and retire the old - DaemonSet honors template nodeSelector against the synthetic node (0 pods when it doesn't match, instead of always one) - CronJob scheduling via TickCronJobs() — materializes a Job from the jobTemplate (no background timer; schedule string stored, driven explicitly) * feat(k8s): server-side apply field ownership + conflict detection - apply patches (application/apply-patch+yaml) track per-fieldManager ownership in metadata.managedFields (real f:-nested FieldsV1) - a conflicting apply by another manager returns 409; force=true transfers ownership; idempotent re-apply by the owner is a no-op - leaf-granularity ownership (map keys / whole arrays); documented subset * feat(k8s): metrics.k8s.io endpoint + HPA actuation - metrics.k8s.io/v1beta1 aggregated API (kubectl top): synthetic Pod/Node metrics from live pods + the synthetic node - HPA reconcile clamps its target Deployment's replicas into [min,max] and reports status - timestamps sourced from the deterministic cluster clock * feat(k8s): enforce ResourceQuota counts, PDB eviction, LimitRange - object-count ResourceQuota enforced on create (403 over limit), status.used updated - pods/eviction subresource honors PodDisruptionBudgets (429 when it would violate) - LimitRange defaulting + min/max validation on pod create - quota reserved only on real (non-dry-run) creates * feat(k8s): RBAC SubjectAccessReview, NetworkPolicy eval, kubectl CI smoke - authorization.k8s.io/v1 SubjectAccessReview evaluated against stored Roles/ClusterRoles + bindings (wildcards, subject matching) - EvaluateNetworkPolicy query API (default-allow; selective deny by ingress from/port) — no live traffic, documented as a query - build-tagged (kubectl) skip-guarded end-to-end smoke test * feat(k8s): admission webhook configs + opt-in admission chain - Mutating/ValidatingWebhookConfiguration kinds (admissionregistration.k8s.io/v1) stored and round-trip via kubectl apply - opt-in admission chain (APIServer.SetAdmissionEnabled): on create/update/patch it calls matching webhooks, applies mutations, and denies (4xx) on reject - admission gates before dry-run/quota so a denied write leaks neither - off by default, honoring the zero-network/deterministic pillar * docs: document full k8s data-plane parity in services.md §18 CRDs, server-side apply, dry-run, finalizers, pagination, pod logs/eviction, metrics.k8s.io + HPA, quota/limitrange/PDB enforcement, RBAC SAR, NetworkPolicy eval, opt-in admission webhooks, watch resume, deterministic clock; rewrite the non-goals into accurate emulation boundaries * docs: refresh k8s data-plane boundaries in sdk-server.md * docs: refresh Kubernetes data-plane summary in README * fix(k8s): resolve data-plane review findings - Finalizers: run CRD onDelete on the finalizer-drain path; owner-GC and namespace cascade mark finalizer-bearing children Terminating instead of reaping them; patch paths no longer resurrect a Terminating object. - ResourceQuota: recompute status.used from the live count on delete; enforce quota on server-side dry-run via a reservation-free check. - Pagination: key-anchored continue token (no skip/dup under mutation) and 410 Gone on a malformed token. - HPA: metric-driven actuation on Resource CPU averageUtilization. - CronJob: real due-evaluation (5-field parser, lastScheduleTime dedup, concurrencyPolicy, startingDeadlineSeconds). - SSA: apply removes omitted owned fields; PUT/PATCH register an Update manager. - Lint: prealloc + G115 fixes; deployment rolling-update test added.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
✨ Features
AWS VPC networking
Azure Cosmos DB for PostgreSQL & Databricks
Microsoft.DBforPostgreSQL/serverGroupsv2) with the Azure SDK/CLI.Microsoft.DatabricksARM resource coverage, so a workspace deployment models end to end.Resource discovery & cost inventory
armresourcegraph(VMs & scale sets, disks, SQL databases/managed instances, storage accounts, Cosmos, AKS, App Service plans, …) and survives real-SDK creation end to end, so a Resource Graph cost query returns the same shape as a real subscription.🔧 Enhancements
AWS service operations & tagging
AddPermission/GetPolicy/RemovePermission), and tagging.GetAuthorizationToken), Secrets Manager update + tagging, IAM inline role policies, ElastiCache cluster modify, CloudWatch Logs retention, and extra EC2 lookups (DescribeRegions,DescribeInstanceTypes, …).Kubernetes data plane
kubectl, Helm, and controller-runtime / operator-SDK workloads behave as they would against a real cluster.Establishedimmediately; deleting it cascade-deletes its custom resources.fieldManagerownership inmanagedFields, 409 conflicts withforcetakeover, and field removal on re-apply.kubectl top(metrics.k8s.io) and CPU-utilization-driven HorizontalPodAutoscaler actuation.🐛 Fixes
AWS
Describe*for a nonexistent ID returns the proper*.NotFoundinstead of an empty result.FilterExpressioninQuery; S3 honorsmax-keys/pagination and servesHeadBucket+ bucket tagging; CloudWatch now works from theawsCLI.GCP
Technical Details
AWS networking (#317, #318, #324)
Networkinginterface is not widened; no Azure/GCP stubs).Kubernetes data plane (#314)
metrics.k8s.io+ HPA actuation, opt-in admission webhooks, RBAC/NetworkPolicy evaluation, ResourceQuota/LimitRange, finalizer-gated deletion, key-anchored list pagination, and deterministic Deployment/DaemonSet/CronJob controllers (Kubernetes data-plane fidelity gaps & recommended additions (follow-ups from PR #299) #312).Azure & resource discovery (#311, #306, #316, #327)
Microsoft.DBforPostgreSQL/serverGroupsv2, Citus) full-parity support.Microsoft.DatabricksARM resources (feat: Azure Databricks remaining ARM resources SDK-compat #209).osType, VMSS, disks, SQL databases/managed instances, storage accounts, Cosmos, AKS, App Service plans), wired through the ARM create paths (Azure Resource Graph: emit fields needed for full downstream cost-discovery E2E (disk IOPS/throughput, SQL DB databases, AKS agent-pool attrs, VMSS) #323).Parity-audit enhancements & fixes (#320, #322)
*.NotFound, DynamoDBFilterExpression, S3 pagination/HeadBucket/tagging, CloudWatch query protocol, plus the additive operations listed under Enhancements (AWS parity E2E audit: bugs, gaps & enhancements across all 30 services #319).@type, non-threshold conditions, and more).